Prevalence of Short Peer Reviews in 3 Leading General Medical Journals.

Importance: High-quality peer reviews are often thought to be essential to ensuring the integrity of the scientific publication process, but measuring peer review quality is challenging. Although imperfect, review word count could potentially serve as a simple, objective metric of review quality. Objective: To determine the prevalence of very short peer reviews and how often they inform editorial decisions on research articles in 3 leading general medical journals. Design, Setting, and Participants: This cross-sectional study compiled a data set of peer reviews from published, full-length original research articles from 3 general medical journals (The BMJ, PLOS Medicine, and BMC Medicine) between 2003 and 2022. Eligible articles were those with peer review data; all peer reviews used to make the first editorial decision (ie, accept vs revise and resubmit) were included. Main Outcomes and Measures: Prevalence of very short reviews was the primary outcome, which was defined as a review of fewer than 200 words. In secondary analyses, thresholds of fewer than 100 words and fewer than 300 words were used. Results were disaggregated by journal and year. The proportion of articles for which the first editorial decision was made based on a set of peer reviews in which very short reviews constituted 100%, 50% or more, 33% or more, and 20% or more of the reviews was calculated. Results: In this sample of 11 466 reviews (including 6086 in BMC Medicine, 3816 in The BMJ, and 1564 in PLOS Medicine) corresponding to 4038 published articles, the median (IQR) word count per review was 425 (253-575) words, and the mean (SD) word count was 520.0 (401.0) words. The overall prevalence of very short (<200 words) peer reviews was 1958 of 11 466 reviews (17.1%). Across the 3 journals, 843 of 4038 initial editorial decisions (20.9%) were based on review sets containing 50% or more very short reviews. The prevalence of very short reviews and share of editorial decisions based on review sets containing 50% or more very short reviews was highest for BMC Medicine (693 of 2585 editorial decisions [26.8%]) and lowest for The BMJ (76 of 1040 editorial decisions [7.3%]). Conclusion and Relevance: In this study of 3 leading general medical journals, one-fifth of initial editorial decisions for published articles were likely based at least partially on reviews of such short length that they were unlikely to be of high quality. Future research could determine whether monitoring peer review length improves the quality of peer reviews and which interventions, such as incentives and norm-based interventions, may elicit more detailed reviews.

Ponderings on peer review. Part 2. Manuscript critiques.

In part 1 of this Perspective, I discussed general principles of scientific peer review in the biomedical sciences aimed at early-stage investigators (i.e., graduate students, postdoctoral fellows, and junior faculty). Here in part 2, I share my thoughts specifically on the topic of peer review of manuscripts. I begin by defining manuscript peer review and discussing the goals and importance of the concept. I then describe the organizational structure of the process, including the two distinct stages involved. Next, I emphasize several important considerations for manuscript reviewers, both general points and key considerations when evaluating specific types of papers, including original research manuscripts, reviews, methods articles, and opinion pieces. I then advance some practical suggestions for developing the written critique document, offer advice for making an overall recommendation to the editor (i.e., accept, revise, reject), and describe the unique issues involved when assessing a revised manuscript. Finally, I comment on how best to gain experience in the essential academic research skill of manuscript peer review. In part 3 of the series, I will discuss the topic of reviewing grant applications submitted to research funding agencies.

Development of a Critical Appraisal Tool (AIMRDA) for the Peer-Review of Studies Assessing the Anticancer Activity of Natural Products: A Step towards Reproducibility.

The journal of APJCP (Asian Pacific Journal of Cancer Prevention) focuses to gather relevant and up-to-date novel information's related to cancer sciences. The research methodologies and approaches adopted by the researcher are prone to variation which may be desirable in the context of novel scientific findings however, the reproducibility for these studies needs to be unified and assured. The reproducibility issues are highly concerned when preclinical studies are reported in cancer, for natural products in particular. The natural products and medicinal plants are prone to a wide variation in terms of phytochemistry and phyto-pharmacology, ultimately affecting the end results for cancer studies. Hence the need for specific guidelines to adopt a best-practice in cancer research are utmost essential. The current AIMRDA guidelines aims to develop a consensus-based tool in order to enhance the quality and assure the reproducibility of studies reporting natural products in cancer prevention. A core working committee of the experts developed an initial draft for the guidelines where more focus was kept for the inclusion of specific items not covered in previous published tools. The initial draft was peer-reviewed, experts-views provided, and improved by a scientific committee comprising of field research experts, editorial experts of different journals, and academics working in different organization worldwide. The feedback from continuous online meetings, mail communications, and webinars resulted a final draft in the shape of a checklist tool, covering the best practices related to the field of natural products research in cancer prevention and treatment. It is mandatory for the authors to read and follow the AIMRDA tool, and be aware of the good-practices to be followed in cancer research prior to any submission to APJCP. Though the tool is developed based on experts in the field, it needs to be further updated and validated in practice via implementation in the field.

Establishing Interrater Reliability Within the Advanced Practice Nurse Peer Review Process.

BACKGROUND AND PURPOSE: The Advanced Practice Nurse (APN) Council refined the APN peer review to an objective, data-driven process. The purpose of the study was to assess the interrater reliability of APN peer reviews using the APN Rubric based on Hamric, Spross & Hanson's Model of Advanced Practice Nursing. METHODS: A quantitative single-site study with a convenience sample of 80 APN Portfolios. RESULTS: Analysis of six core competencies (direct clinical practice, leadership, consultation/collaboration, coaching/guiding, research, and ethical decision-making) within the APN Rubric demonstrated substantial and near perfect agreement levels in the APN peer review process. CONCLUSIONS: The application of APN core competencies within the peer review process demonstrated high consistency, thereby increasing the significance and objectivity of peer review outcomes.

Customized guidance/training improved the psychometric properties of methodologically rigorous risk of bias instruments for non-randomized studies.

OBJECTIVES: To evaluate the impact of guidance and training on the inter-rater reliability (IRR), inter-consensus reliability (ICR) and evaluator burden of the Risk of Bias (RoB) in Non-randomized Studies (NRS) of Interventions (ROBINS-I) tool, and the RoB instrument for NRS of Exposures (ROB-NRSE). STUDY DESIGN AND SETTING: In a before-and-after study, seven reviewers appraised the RoB using ROBINS-I (n = 44) and ROB-NRSE (n = 44), before and after guidance and training. We used Gwet's AC1 statistic to calculate IRR and ICR. RESULTS: After guidance and training, the IRR and ICR of the overall bias domain of ROBINS-I and ROB-NRSE improved significantly; with many individual domains showing either a significant (IRR and ICR of ROB-NRSE; ICR of ROBINS-I), or nonsignificant improvement (IRR of ROBINS-I). Evaluator burden significantly decreased after guidance and training for ROBINS-I, whereas for ROB-NRSE there was a slight nonsignificant increase. CONCLUSION: Overall, there was benefit for guidance and training for both tools. We highly recommend guidance and training to reviewers prior to RoB assessments and that future research investigate aspects of guidance and training that are most effective.

Authors, peer reviewers, and readers: What is expected from each player in collaborative publishing?

Scholarly publishing is in a crisis, with the many stakeholders complaining about different aspects of the system. Authors want fast publication times, high visibility and publications in high-impact journals. Readers want freely accessible, high-quality articles. Peer reviewers want recognition for the work they perform to ensure the quality of the published articles. However, authors, peer reviewers, and readers are three different roles played by the same group of individuals, the users of the scholarly publishing system-and this system could work based on a collaborative publishing principle where "nobody pays, and nobody gets paid"

No disponible

Publish or Perish: Five Steps to Navigating a Less Painful Peer Review.

This Perspective presents comments intended for junior researchers by Carol A. Lange, Editor-in-Chief, Endocrinology, and Stephen R. Hammes, former Editor-in-Chief, Molecular Endocrinology, and former co-Editor-in-Chief, Endocrinology. PRINCIPAL POINTS: 1. Know when you are ready and identify your target audience.2. Select an appropriate journal.3. Craft your title and abstract to capture your key words and deliver your message.4. Tell a clear and impactful story.5. Review, polish, and perfect your manuscript.

Effects of Knowledge and Value on Quality of Supportive Peer Feedback.

BACKGROUND: Motivational barriers and lack of knowledge about peer review inhibit creation of supportive peer feedback between nursing students. PURPOSE: The purpose of this study was to examine the effects of knowledge of peer-review practices and value for the process of creating supportive peer feedback on the quality of feedback nursing students create for their peers. METHOD: Data from 155 graduate nursing students were analyzed in this quasi-experimental, repeated-measures design, using analysis of covariance and conditional process analysis. RESULTS: Students who had high levels of knowledge and a strong sense of value for providing feedback to peers produced higher-quality supportive peer feedback. CONCLUSION: Even when a student has sufficient knowledge and skills to produce supportive peer feedback, their ability may not transfer to real peer-review contexts if they do not also have a strong sense of value for providing high-quality feedback to peers.

A novel faculty development tool for writing a letter of recommendation.

OBJECTIVE: Based on a national survey of program directors we developed a letter of recommendation (LOR) scoring rubric (SR) to assess LORs submitted to a pediatric residency program. The objective was to use the SR to analyze: the consistency of LOR ratings across raters and LOR components that contributed to impression of the LOR and candidate. METHODS: We graded 30 LORs submitted to a pediatric residency program that were evenly distributed based on final rank by our program. The SR contained 3 sections (letter features, phrases, and applicant abilities) and 2 questions about the quality of the LOR (LORQ) and impression of the candidate (IC) after reading the LOR on a 5-point Likert scale. Inter-rater reliability was calculated with intraclass correlation coefficients (ICC(2,1)). Pearson (r) correlations and stepwise multivariate linear regression modeling predicted LORQ and IC. Mean scores of phrases, features, and applicant abilities were analyzed with ANOVA and Bonferroni correction. RESULTS: Phrases (ICC(2,1) = 0.82, p<0.001)) and features (ICC(2,1) = 0.60, p<0.001)) were rated consistently, while applicant abilities were not (ICC(2,1) = 0.28, p<0.001)). For features, LORQ (R2 = 0.75, p<0.001) and IC (R2 = 0.58, p<0.001) were best predicated by: writing about candidates' abilities, strength of recommendation, and depth of interaction with the applicant. For abilities, LORQ (R2 = 0.47, p<0.001) and IC (R2 = 0.51, p<0.001) were best predicted by: clinical reasoning, leadership, and communication skills (0.2). There were significant differences for phrases and features (p<0.05). CONCLUSIONS: The SR was consistent across raters and correlates with impression of LORQ and IC. This rubric has potential as a faculty development tool for writing LORS.